Presentation of communications

ABSTRACT

A method to present communications may include captioning, by a human assistant during a call between a first user using a first captioning telephone device and a second user using a second telephone device, words spoken by the second user into the second telephone device. The method may also include presenting the captioned words on a first display of the first captioning telephone device, receiving text typed into the second telephone device by the second user, and presenting the received text on the first display of the first captioning telephone device.

FIELD

The embodiments discussed in the present disclosure are related topresentation of communications.

BACKGROUND

Transcriptions of audio communications may assist people that arehearing-impaired to participate in the audio communications. To providethe transcriptions to a hearing-impaired person, a hearing-impaired usermay participate in a captioning communication session with a hearinguser in which a human assistant transcribes words spoken by the hearinguser. In this manner, the hearing-impaired user may listen to the wordsspoken by the hearing user as well as read captions of the words spokenby the hearing user.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one example technology area where some embodiments describedherein may be practiced.

SUMMARY

According to one or more aspects of the present disclosure, a method topresent communications may include captioning, by a human assistantduring a call between a first user using a first captioning telephonedevice and a second user using a second telephone device, words spokenby the second user into the second telephone device. The method may alsoinclude presenting the captioned words on a first display of the firstcaptioning telephone device, receiving text typed into the secondtelephone device by the second user, and presenting the received text onthe first display of the first captioning telephone device.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates an example environment for presentation ofcommunications;

FIG. 2 illustrates an example screen that may appear on a display of adevice associated with presentation of communications;

FIG. 3 illustrates example operations related to presentation ofcommunications;

FIG. 4 illustrates a flowchart of an example method to presentcommunications; and

FIG. 5 illustrates an example system that may be used duringpresentation of communications.

DESCRIPTION OF EMBODIMENTS

Some embodiments in this disclosure relate to systems and methods thatmay be configured to present communications. For example, in someembodiments, a hearing-impaired user may participate in a captioningcommunication session with a hearing user in which a human assistanttranscribes words spoken by the hearing user. In this manner, thehearing-impaired user may listen to the words spoken by the hearing useras well as read captions of the words spoken by the hearing user.

One problem with the transcription of words spoken by a hearing userduring a captioning communication session may involve miscommunicationdue to the hearing user not speaking words clearly or due to the spokenwords being difficult to understand. For example, where the hearing useris speaking English, but English is not the hearing user's nativelanguage, it may be difficult for the hearing-impaired user and for thehuman assistant to understand the English words spoken by the hearinguser (or other non-English words spoken by the hearing user), which maymake it difficult for the human assistant to accurately transcribe thewords spoken by the hearing user for use by the hearing-impaired user.

In some embodiments, during a call between a hearing-impaired user usinga captioning telephone device and a hearing user using a telephonedevice, the hearing-impaired user may be presented with captions, on adisplay of the captioning telephone device, of the words spoken by thehearing user. In addition, the hearing-impaired user may be presented,on the display of the captioning telephone device, with text typed intothe second telephone device by the hearing user. In this manner, thehearing user may have the option to either speak words that will becaptioned for the hearing-impaired user or type words that will bepresented to the hearing-impaired user, thus avoiding miscommunicationby enabling words that may be difficult to speak clearly or that may bedifficult to understand (for the hearing-impaired user and/or for thehuman assistant) to alternatively or additionally be precisely typed bythe hearing user and presented to the hearing-impaired user.

Turning to the figures, FIG. 1 illustrates an example environment 100for presentation of communications. The environment 100 may be arrangedin accordance with at least one embodiment described in the presentdisclosure. The environment 100 may include a network 102, a firstdevice 104, a second device 106, and a transcription system 108.

The network 102 may be configured to communicatively couple the firstdevice 104, the second device 106, and the transcription system 108. Insome embodiments, the network 102 may be any network or configuration ofnetworks configured to send and receive communications between systemsand devices. In some embodiments, the network 102 may include a wirednetwork, an optical network, and/or a wireless network, and may havenumerous different configurations, including multiple different types ofnetworks, network connections, and protocols to communicatively coupledevices and systems in the environment 100. In some embodiments, thenetwork 102 may also be coupled to or may include portions of atelecommunications network, including telephone lines, for sending datain a variety of different communication protocols, such as a plain oldtelephone system (POTS).

Each of the first and second devices 104 and 106 may be any electronicor digital computing device. For example, each of the first and seconddevices 104 and 106 may include a desktop computer, a laptop computer, asmartphone, a mobile phone, a tablet computer, a telephone, a phoneconsole, a caption device, a captioning telephone device, a dual-tonemulti-frequency signaling (DTMF) telephone device, a voice-over Internetprotocol (VOIP) telephone device, or any other computing device that maybe used for communication between a hearing-impaired user 110 using thefirst device 104 and a hearing user 112 using the second device 106.

In some embodiments, each of the first and second devices 104 and 106may include memory and at least one processor, which are configured toperform operations as described in this disclosure, among otheroperations. In some embodiments, each of the first and second devices104 and 106 may include computer-readable instructions that areconfigured to be executed by each of the first and second devices 104and 106 to perform operations described in this disclosure, among otheroperations.

In some embodiments, the first and second devices 104 and 106 may beconfigured to establish communication sessions with other devices. Forexample, the first device 104 may be configured to establish an outgoingcommunication session, such as a telephone call, video call, or othercommunication session, with another device over a telephone line ornetwork. For example, the first device 104 and the second device 106 maycommunicate with the network 102 over a wireless cellular network, awired Ethernet network, a wireless 802.11 network, an optical network,or a POTS line. Alternatively or additionally, each of the first device104 and the second device 106 may communicate over other wired orwireless networks that do not include or only partially include a POTS.For example, a communication session between the first device 104 andthe second device 106, such as a telephone call, may be a VoIP telephonecall. As another example, the communication session between the firstdevice 104 and the second device 106 may be a video communicationsession or other communication session.

Alternately or additionally, each of the first and second devices 104and 106 may be configured to communicate with other systems over anetwork, such as the network 102 or another network. In these and otherembodiments, each of the first device 104 and the second device 106 mayreceive data from and send data to the transcription system 108.

In some embodiments, the first and second devices 104 and 106 may beconfigured to obtain audio during a communication session. The audio maybe part of a video communication or an audio communication, such as atelephone call. As used in this disclosure, the term audio may be usedgenerically to refer to sounds that may include spoken words.Furthermore, the term “audio” may be used generically to include audioin any format, such as a digital format, an analog format, or apropagating wave format. Also, as used in this disclosure, the termvideo may be used generically to refer to a compilation of images thatmay be reproduced in a sequence to produce video.

As an example of obtaining audio, the first device 104 may be configuredto obtain first audio from the hearing-impaired user 110. For example,the first device 104 may obtain the first audio from a microphone 122 ofthe first device 104 or from another device that is communicativelycoupled to the first device 104. Similarly, the second device 106 mayalso be configured to obtain second audio from the hearing user 112. Insome embodiments, the second device 106 may obtain the second audio froma microphone 128 of the second device 106 or from another devicecommunicatively coupled to the second device 106. During thecommunication session, the first device 104 may provide the first audioto the second device 106 for broadcasting on a speaker 126 of the seconddevice 106. Alternatively or additionally, the second device 106 mayprovide the second audio to the first device 104 for broadcasting on aspeaker 120 of the first device 104.

In some embodiments, the transcription system 108 may include anyconfiguration of hardware, such as processors, servers, and databaseservers that are networked together and configured to perform a task.For example, the transcription system 108 may include multiple computingsystems, such as multiple servers that each include memory and at leastone processor, which are networked together and configured to performoperations of captioning communication sessions, such as telephonecalls, between devices such as the first device 104 and another device(e.g., the second device 106) as described in this disclosure. In theseand other embodiments, the transcription system 108 may operate togenerate transcriptions of audio of one or more parties in acommunication session. For example, the transcription system 108 maygenerate transcriptions of audio generated by other devices and not thefirst device 104 or both the first device 104 and other devices, amongother configurations.

In some embodiments, the transcription system 108 may operate as anexchange configured to establish communication sessions, such astelephone calls, video calls, etc., between devices such as the firstdevice 104 and another device or devices as described in thisdisclosure, among other operations. In some embodiments, thetranscription system 108 may include computer-readable instructions thatare configured to be executed by the transcription system 108 to performoperations described in this disclosure.

Further, in some embodiments, the environment 100 may be configured tofacilitate a captioning communication session between thehearing-impaired user 110 and a second user, such as the hearing user112. As used in the present disclosure, a “hearing-impaired user” mayrefer to a person with diminished hearing capabilities. Hearing-impairedusers often have some level of hearing ability that has usuallydiminished over a period of time such that the hearing-impaired user cancommunicate by speaking, but the hearing-impaired user may struggle inhearing and/or understanding others.

In some embodiments, the captioning communication session may beestablished between the first device 104 and the second device 106. Inthese embodiments, the first device 104 may be configured to presenttranscriptions of the communication session to the hearing-impaired user110. As an example, the first device 104 may be one of the CaptionCall®57T model family or 67T model family of captioning telephones or adevice running the CaptionCall® mobile app. For example, in someembodiments, the first device 104 may include a visual display 116, suchas a touchscreen visual display or other visual display, that isintegral with the first device 104 and that is configured to presenttext transcriptions of a communication session to the hearing-impaireduser 110.

Alternatively or additionally, the first device 104 may be associatedwith a visual display that is physically separate from the first device104 and that is in wireless communication with the first device 104,such as a visual display of a smartphone accessible to thehearing-impaired user 110 and configured to be in Bluetooth® wirelesscommunication with the first device 104. Other physically separatephysical displays may be visual displays of desktop computers, laptopcomputers, mobile phones, tablet computers, or any other computingdevices that are in wireless communication with the first device 104.

The first device 104 may also include the speaker 120, such as a speakerin a handset or a speaker in a speakerphone. The first device 104 mayalso include a processor communicatively coupled to the visual display116 and to the speaker 120, as well as at least one non-transitorycomputer-readable media communicatively coupled to the processor andconfigured to store one or more instructions that when executed by theprocessor perform the methods for presentation of messages as describedin this disclosure, and also store voice messages locally on the firstdevice 104.

During a communication session, the transcription system 108, the firstdevice 104, and the second device 106 may be communicatively coupledusing networking protocols. In some embodiments, during thecommunication session between the first device 104 and the second device106, the first device 104 may provide the audio received from the seconddevice 106 to the transcription system 108. Alternatively oradditionally, the second device 106 may provide the audio to thetranscription system 108 and the transcription system 108 may relay theaudio to the first device 104. Alternatively or additionally, video datamay be provided to the transcription system 108 from the second device106 and relayed to the first device 104.

At the transcription system 108, the audio data may be transcribed. Insome embodiments, to transcribe the audio data, a transcription enginemay generate a transcription of the audio. For example, thetranscription system 108 may be configured to generate transcriptions ofaudio obtained from devices with which the transcription system 108 isassociated. The transcription system 108 may be configured to generatethe transcriptions of the audio in real-time or substantially real-timeduring a communication session and send the transcriptions to theassociated devices. For example, the first device 104 may send secondaudio that originates from the second device 106 to the transcriptionsystem 108 during a communication session with the second device 106.The transcription system 108 may generate a transcription of the secondaudio and provide the transcription to the first device 104. The firstdevice 104 may present the transcription during the communicationsession in real-time or substantially real-time. In some embodiments,real-time or substantially real-time may indicate that a time differencebetween when words are audibly broadcast by the first device 104 andcorresponding words are presented in the transcription by the firstdevice 104 may be less than five, ten, fifteen, or twenty seconds.

The transcription system 108 may generate transcriptions using a fullymachine-based automatic speech recognition (ASR) process that mayoperate without human intervention. Alternately or additionally, thetranscription system 108 may be configured to generate a transcriptionof audio using a revoicing process. The revoicing process may receiveand broadcast audio to a human assistant 114. The human assistant 114may listen to the broadcast and speak the words from the broadcast. Thewords spoken by the human assistant 114 may be captured to generaterevoiced audio. The revoiced audio may be used by a speech recognitionprogram to generate the transcription of the audio. For example, thehuman assistant 114 may listen to the audio received from the seconddevice 106 at the transcription system 108 (e.g., received via the firstdevice 104), and “revoice” the words of the hearing user 112 to a speechrecognition computer program tuned to the voice of the human assistant114. In these and other embodiments, the human assistant 114 may be anoperator who serves as a human intermediary between the hearing-impaireduser 110 and the hearing user 112. In some embodiments, texttranscriptions may be generated by a speech recognition computer as atranscription of the audio of the hearing user 112. Alternately oradditionally, the transcription system 108 may use a combination of amachine-based process and a revoicing process to generate atranscription of the audio.

After generation of the text transcriptions, the text transcriptions maybe provided to the first device 104 over the network 102. The firstdevice 104 may display the text transcriptions on the visual display 116while the hearing-impaired user 110 carries on a normal voiceconversation with the hearing user 112. The text transcriptions mayallow the hearing-impaired user 110 to supplement the voice signalreceived from the second device 106 and confirm her understanding of thewords spoken by the hearing user 112. The transcription of acommunication session occurring in real-time between two devices asdiscussed above may be referred to in this disclosure as a captioningcommunication session.

In some embodiments, the transcription system 108 may be associated withone or more of the first and second devices 104 and 106. A device beingassociated with the transcription system 108 may indicate that thedevice includes credentials to allow the device to communicate with andreceive transcription services from the transcription system 108. Forexample, a device associated with the transcription system 108 mayindicate that the device may send audio to the transcription system 108and obtain a transcription of the audio from the transcription system108.

In addition to generating transcriptions of communication sessions, theenvironment 100 may be configured to provide typed text from otherdevices, such as the second device 106. The typed text may include anycharacters that may be typed into a physical or digital keypad of adevice, such as alphabetic characters, numeric characters, emojicharacters, or other characters. In some embodiments, during acaptioning communication session, the hearing user 112 may type textinto a digital keypad on the touchscreen display 124 of the seconddevice 106, such as into a digital keypad of a captioning app running onthe second device 106. This typed text may then be sent to the firstdevice 104 and presented to the hearing-impaired user 110 on the visualdisplay 116. Additionally or alternatively, the hearing-impaired user110 may type text into a physical keypad 118, or into a digital keypadon the visual display 116. This typed text may then be sent to thesecond device 106 and presented to the hearing user 112 in thecaptioning app on the touchscreen display 124. Alternatively, this typedtext may be converted (e.g., on the first device 104, on the seconddevice 106, or on the transcription system 108) into audio using atext-to-speech conversion, and the audio may then be broadcast on thespeaker 126 of the second device 106. In this manner, the hearing user112 may have the option to either speak words that will be captioned forthe hearing-impaired user 110 or type words that will be presented tothe hearing-impaired user 110, thus avoiding miscommunication byenabling words that may be difficult to speak clearly or that may bedifficult to understand (for the hearing-impaired user 110 and/or forthe human assistant 114) to alternatively or additionally be preciselytyped by the hearing user 112 and presented to the hearing-impaired user110.

Modifications, additions, or omissions may be made to the environment100 without departing from the scope of the present disclosure. Forexample, in some embodiments, the user 112 may be a hearing-impaireduser and/or the user 110 may be a hearing user. Alternately oradditionally, the transcription system 108 may provide text to thesecond device 106 based on audio transmitted by the first device 104.Alternately or additionally, the transcription system 108 may includeadditional functionality. For example, the transcription system 108 mayedit the text or make other alterations to the text after presentationof the text on the first device 104. Alternately or additionally, theenvironment 100 may include additional devices similar to the first andsecond devices 104 and 106. In these and other embodiments, theseadditional devices may be configured to present communications asdescribed in this disclosure.

FIG. 2 illustrates an example screen 200 that may appear on a display ofa device associated with presentation of communications, arranged inaccordance with at least one embodiment described in the presentdisclosure. The screen 200 of FIG. 2 may present captioned text 202, asprovided in real-time or substantially real-time by a transcriptionsystem (e.g., the transcription system 108 of FIG. 1 ), as well as typedtext 204, as provided in real-time or substantially real-time from auser typing text into a device (e.g., the hearing user 112 typing textinto the second device 106 of FIG. 1 ). Additionally, the captioned text202 may be presented with an indicator 206 that indicates that thecaptioned words were spoken by a user (e.g., the hearing user 112 ofFIG. 1 ), and the typed text 204 may be presented with an indicator 208that indicates that the typed words were typed by the user. Although theindicators 206 and 208 are illustrated in FIG. 2 as text labels, it isunderstood that these indicators may instead be symbols, pictures, orany other indicator that allows captioned text to be distinguished fromthe typed text. Modifications, additions, or omissions may be made tothe screen 200 without departing from the scope of the presentdisclosure.

FIG. 3 illustrate example operations 300 related to presentation ofcommunications. The operations 300 may be arranged in accordance with atleast one embodiment described in the present disclosure. In theillustrated example, the operations 300 may be between the first device104, the second device 106, and the transcription system 108 of FIG. 1 .In some embodiments, the operations 300 may be an example ofcommunications and interactions between the first device 104, the seconddevice 106, and the transcription system 108. In some embodiments, theinteractions between the first device 104, the second device 106, andthe transcription system 108 may occur over a first network (e.g., thenetwork 102 of FIG. 1 ). Generally, the operations 300 may relate topresentation of communications between the first device 104 and thesecond device 106. The operations 300 illustrated are not exhaustive butare merely representative of operations 300 that may occur. Furthermore,one operation as illustrated may represent one or more communications,operations, and/or data exchanges.

At operation 302, a communication session may be established between thefirst device 104 and the second device 106. During the communicationsession, audio obtained by the second device 106 may be directed to thefirst device 104. Alternately or additionally, audio obtained by thefirst device 104 may be directed to the second device 106. Alternatelyor additionally, text typed into the second device 106 may be presentedon the first device 104. Alternately or additionally, text typed intothe first device 104 may be presented on the second device 106.

At operation 304, the second device 106 may direct audio (e.g.,including words spoken by a hearing user) to the first device 104. Thesecond device 106 may direct audio to the first device 104 using thenetwork and the communication protocols for the communication sessionthat were established during the operation 302.

At operation 306, the first device 104 may be configured to audiblybroadcast the audio of the communication session, such as on the speaker120 of the first device 104.

At operation 308, the first device 104 may be configured to direct theaudio to the transcription system 108. The first device 104 may directthe audio to the transcription system 108 based on a transcriptionsystem protocol included in the first device 104. In these and otherembodiments, the first device 104 may be elected to send the audio tothe transcription system 108 based on the first device 104 beingassociated with the transcription system 108. To direct the audio to thetranscription system 108, the first device 104 may establish a networkconnection with the transcription system 108 using networking protocols,passwords, or other procedures to establish the network connection.After establishing a network connection, a transcription session may beestablished by the transcription system 108 to generate a transcriptionof the audio. The transcription session may be between the first device104 and the transcription system 108.

At operation 310, the transcription system 108 may transcribe the audio.This transcription of the audio may be accomplished using a humanassistant (e.g., the human assistant 114).

At operation 312, the transcription system 108 may direct thetranscription of the audio generated by the transcription session to thefirst device 104. The transcription may be directed using the networkand the communication protocols for the transcription session that wereestablished during the operation 308.

At operation 314, the first device 104 may be configured to display thetranscription, such as on the visual display 116 of the first device104. The first device 104 may be configured to display the transcriptionin substantially real-time or real-time in configuration with audiblybroadcasting the remote audio in operation 306. In this manner, thehearing-impaired user 110 may simultaneously (or nearly simultaneously)hear words spoken by the hearing user 112 as well as read captions ofthe words spoken by the hearing user 112. In some embodiments, thetranscription may be displayed with an indicator (e.g., the indicator206) that indicates that the received text was spoken by the hearinguser 112.

At operation 316, the second device 106 may direct typed text to thefirst device 104. The second device 106 may direct the typed text to thefirst device 104 using the network and the communication protocols forthe communication session that were established during the operation302. The text may be typed via a touchscreen, a physical keyboard,handwriting recognition, or some form of touchscreen-based swipekeyboard. For example, where the second device 106 is a smartphonedevice, the text may be typed by the hearing user 112 into a captioningapp via the touchscreen display 124 of the smartphone device. In anotherexample, where the second device 106 is a VoIP telephone device, thetext may be typed by the hearing user 112 into a captioning app via adisplay of the VoIP telephone device. In another example, where thesecond device 106 is a DTMF telephone device, the text may be typed bythe hearing user 112 into a telephone keypad of the DTMF telephonedevice, and then converted into a DTMF signal that can be decoded by thefirst device 104. In another example, where the second device 106 is adesktop computer, the text may be typed by the hearing user 112 via akeyboard of the desktop computer into webpage in a browser on a displayof the desktop computer.

At operation 318, the first device 104 may be configured to present thetyped text on the first device 104, such as on the visual display 116 ofthe first device 104. In some embodiments, the typed text may bedisplayed with an indicator (e.g., the indicator 208) that indicatesthat the typed text was typed by the hearing user 112.

At operation 320, the first device 104 may be configured to direct thetyped text to the transcription system 108. The first device 104 maydirect the typed text to the transcription system 108 based on atranscription system protocol included in the first device 104 over thetranscription session established during the operation 308.

At operation 322, the transcription system 108 may be configured topresent the typed text on a display of a computing device of a humanassistant (e.g., the human assistant 114 of FIG. 1 ). This typed textmay be presented to the human assistant 114 to enable the humanassistant 114 to more easily follow the conversation, which may enablethe human assistant 114 to more accurately transcribe spoken words thatfollow the typed text.

At operation 324, the first device 104 may direct audio (e.g., includingwords spoken by the hearing-impaired user 110) to the second device 106.The first device 104 may direct audio to the second device 106 using thenetwork and the communication protocols for the communication sessionthat were established during the operation 302.

At operation 326, the second device 106 may be configured to audiblybroadcast the audio of the communication session on the second device106, such as on the speaker 126 of the second device 106.

At operation 328, the first device 104 may direct typed text to thesecond device 106. The first device 104 may direct the typed text to thesecond device 106 using the network and the communication protocols forthe communication session that were established during the operation302. For example, the text may be typed by the hearing-impaired user 110into the physical keypad 118 of the first device 104, or may be typeinto a digital keypad presented on the visual display 116 of the firstdevice 104.

At operation 330, the second device 106 may be configured to present thetyped text on the second device 106, such as on the touchscreen display124 via a captioning app of the second device 106. For example, wherethe second device 106 is a smartphone device, the typed text may bepresented on captioning app on the touchscreen display 124 of thesmartphone device. In another example, where the second device 106 is aVoIP telephone device, the typed text may be presented in captioning appon a display of the VoIP telephone device. In another example, where thesecond device 106 is a DTMF telephone device, the type text may beconverted into audio using a text-to-speech conversion, and thenbroadcast on a speaker of the DTMF telephone device. The text-to-speechconversion may operate in word mode where words are articulated orcharacter mode where individual characters are articulated, which may beuseful for spelling, providing numerical data, or articulate charactersof words that are difficult to pronunciate. In another example, wherethe second device 106 is a desktop computer, the typed text may bepresented in a webpage in a browser on a display of the desktopcomputer.

Modifications, additions, or omissions may be made to the operations 300without departing from the scope of the present disclosure. For example,the operations 300 may not include the operations 320 and 322 in someembodiments. As another example, the operations 300 may not include theoperations 324, 326, 328, and 330 in some embodiments. Further, in someembodiments the operations 300 may be arranged in a different order orperformed at the same time. For example, the operations 316 and 318 mayoccur before the operations 304, 306, 308, 310, and 312. Alternately oradditionally, the operations 316 and 318 may occur at the same time orin substantially overlapping time periods as the operations 304, 306,308, 310, and 312.

FIG. 4 illustrates a flowchart of an example method 400 to presentcommunications. The method 400 may be arranged in accordance with atleast one embodiment described in the present disclosure. One or moreoperations of the method 400 may be performed, in some embodiments, by adevice or system, such as the first device 104 or the second device 106of FIG. 1 or the system 500 of FIG. 5 or another device, combination ofdevices, or system. In these and other embodiments, the method 400 maybe performed based on the execution of instructions stored on one ormore non-transitory computer-readable media. Although illustrated asdiscrete blocks, various blocks may be divided into additional blocks,combined into fewer blocks, or eliminated, depending on the desiredimplementation.

The method 400 may include, at block 402, captioning words spoken by asecond user into a second telephone device. In some embodiments, thecaptioning may be performed by a human assistant. In some embodiments,the captioning may be performed during a call between a first user usinga first captioning telephone device and the second user using the secondtelephone device. In some embodiments, the first user may be ahearing-impaired user. In some embodiments, the second user may be ahearing user.

The method 400 may include, at block 404, presenting the captioned wordson a first display of a first captioning telephone device.

The method 400 may include, at block 406, receiving text typed into thesecond telephone device by the second user.

The method 400 may include, at block 408, presenting the received texton the first display of the first captioning telephone device. In someembodiments, the presenting of the received text on the first display ofthe first captioning telephone device may include presenting thereceived text on the first display of the first captioning telephonedevice with a first indicator that indicates that the received text wastyped by the second user. In some embodiments, the presenting of thecaptioned words on the first display of the first captioning telephonedevice may include presenting the captioned words on the first displayof the first captioning telephone device with a second indicator thatindicates that the captioned words were spoken by the second user.

In some embodiments, the second telephone device may include asmartphone device having a touchscreen display and running a captioningapp. In some embodiments, the text typed into the smartphone device mayinclude text typed into the captioning app via the touchscreen display.In some embodiments, the method 400 may also include receiving secondtext typed into the first captioning telephone device by the first user,and presenting the second text in the captioning app on the touchscreendisplay of the smartphone device.

In some embodiments, the second telephone device the second telephonedevice may include a VoIP telephone device having a second display andrunning a captioning app. In some embodiments, the text typed into theVoIP telephone device may include text typed into the captioning app. Insome embodiments, the method 400 may further include receiving secondtext typed into the first captioning telephone device by the first user,and presenting the received second text in the captioning app on thesecond display of the VoIP telephone device.

In some embodiments, the second telephone device may include a DTMFtelephone device including a telephone keypad and a speaker. In someembodiments, the text typed into the DTMF telephone device by the seconduser may include text typed into the telephone keypad. In someembodiments, the method 400 may further include receiving second texttyped into the first captioning telephone device by the first user,converting the second text into audio using a text-to-speech conversion,and sending the audio to the speaker of the DTMF telephone device.

It is understood that, for this and other processes, operations, andmethods disclosed herein, the functions and/or operations performed maybe implemented in differing order. Furthermore, the outlined functionsand operations are only provided as examples, and some of the functionsand operations may be optional, combined into fewer functions andoperations, or expanded into additional functions and operations withoutdetracting from the essence of the disclosed embodiments. For example,in some embodiments, the method 400 may further include presenting thereceived text on a display of a computing device of the human assistant.

FIG. 5 illustrates an example system 500 that may be used duringtranscription of communications. The system 500 may be arranged inaccordance with at least one embodiment described in the presentdisclosure. The system 500 may include a processor 510, memory 512, acommunication system 516, a display 518, a user interface 520, and aperipheral device 522, which all may be communicatively coupled. In someembodiments, the system 500 may be part of any of the systems or devicesdescribed in this disclosure.

For example, the system 500 may be part of the first device 104 of FIG.1 and may be configured to perform one or more of the tasks describedabove with respect to the first device 104. As another example, thesystem 500 may be part of the second device 106 of FIG. 1 and may beconfigured to perform one or more of the tasks described above withrespect to the second device 106. As another example, the system 500 maybe part of the transcription system 108 of FIG. 1 and may be configuredto perform one or more of the tasks described above with respect to thetranscription system 108.

Generally, the processor 510 may include any suitable special-purpose orgeneral-purpose computer, computing entity, or processing deviceincluding various computer hardware or software modules and may beconfigured to execute instructions stored on any applicablecomputer-readable storage media. For example, the processor 510 mayinclude a microprocessor, a microcontroller, a parallel processor suchas a graphics processing unit (GPU) or tensor processing unit (TPU), adigital signal processor (DSP), an application-specific integratedcircuit (ASIC), a Field-Programmable Gate Array (FPGA), or any otherdigital or analog circuitry configured to interpret and/or to executeprogram instructions and/or to process data.

Although illustrated as a single processor in FIG. 5 , it is understoodthat the processor 510 may include any number of processors distributedacross any number of networks or physical locations that are configuredto perform individually or collectively any number of operationsdescribed herein. In some embodiments, the processor 510 may interpretand/or execute program instructions and/or process data stored in thememory 512. In some embodiments, the processor 510 may execute theprogram instructions stored in the memory 512.

For example, in some embodiments, the processor 510 may execute programinstructions stored in the memory 512 that are related to transcriptionof communications such that the system 500 may perform or direct theperformance of the operations associated therewith as directed by theinstructions. In these and other embodiments, the instructions may beused to perform one or more operations 300 of the FIG. 3 or the method400 of FIG. 4 .

The memory 512 may include computer-readable storage media or one ormore computer-readable storage mediums for carrying or havingcomputer-executable instructions or data structures stored thereon. Suchcomputer-readable storage media may be any available media that may beaccessed by a general-purpose or special-purpose computer, such as theprocessor 510.

By way of example, and not limitation, such computer-readable storagemedia may include non-transitory computer-readable storage mediaincluding Random Access Memory (RAM), Read-Only Memory (ROM),Electrically Erasable Programmable Read-Only Memory (EEPROM), CompactDisc Read-Only Memory (CD-ROM) or other optical disk storage, magneticdisk storage or other magnetic storage devices, flash memory devices(e.g., solid state memory devices), or any other storage medium whichmay be used to carry or store particular program code in the form ofcomputer-executable instructions or data structures and which may beaccessed by a general-purpose or special-purpose computer. Combinationsof the above may also be included within the scope of computer-readablestorage media.

Computer-executable instructions may include, for example, instructionsand data configured to cause the processor 510 to perform a certainoperation or group of operations as described in this disclosure. Inthese and other embodiments, the term “non-transitory” as explained inthe present disclosure should be construed to exclude only those typesof transitory media that were found to fall outside the scope ofpatentable subject matter in the Federal Circuit decision of In reNuuten, 500 F.3d 1346 (Fed. Cir. 2007). Combinations of the above mayalso be included within the scope of computer-readable media.

The communication system 516 may include any component, device, system,or combination thereof that is configured to transmit or receiveinformation over a network. In some embodiments, the communicationsystem 516 may communicate with other devices at other locations, thesame location, or even other components within the same system. Forexample, the communication system 516 may include a modem, a networkcard (wireless or wired), an infrared communication device, a wirelesscommunication device (such as an antenna), and/or chipset (such as aBluetooth® device, an 802.6 device (e.g., Metropolitan Area Network(MAN)), a WiFi device, a WiMax device, cellular communicationfacilities, etc.), and/or the like. The communication system 516 maypermit data to be exchanged with a network and/or any other devices orsystems described in the present disclosure. For example, when thesystem 500 is included in the first device 104 of FIG. 1 , thecommunication system 516 may allow the first device 104 to communicatewith the transcription system 108.

The display 518 may be configured as one or more displays, like an LCD,LED, Braille terminal, or other type of display. The display 518 may beconfigured to present video, text captions, user interfaces, and otherdata as directed by the processor 510. For example, when the system 500is included in the first device 104 of FIG. 1 , the display 518 may beconfigured to present a transcription of second audio from the seconddevice 106.

The user interface 520 may include any device to allow a user tointerface with the system 500. For example, the user interface 520 mayinclude a mouse, a track pad, a keyboard, buttons, camera, and/or atouchscreen, among other devices. The user interface 520 may receiveinput from a user and provide the input to the processor 510. In someembodiments, the user interface 520 and the display 518 may be combined.

The peripheral devices 522 may include one or more devices. For example,the peripheral devices may include a microphone, an imager, and/or aspeaker, among other peripheral devices. In these and other embodiments,the microphone may be configured to capture audio. The imager may beconfigured to capture images. The images may be captured in a manner toproduce video or image data. In some embodiments, the speaker maybroadcast audio received by the system 500 or otherwise generated by thesystem 500.

Modifications, additions, or omissions may be made to the system 500without departing from the scope of the present disclosure. For example,in some embodiments, the system 500 may include any number of othercomponents that may not be explicitly illustrated or described. Further,depending on certain implementations, the system 500 may not include oneor more of the components illustrated and described.

As indicated above, the embodiments described herein may include the useof a special purpose or general-purpose computer (e.g., the processor510 of FIG. 5 ) including various computer hardware or software modules,as discussed in greater detail below. Further, as indicated above,embodiments described herein may be implemented using computer-readablemedia (e.g., the memory 512 of FIG. 5 ) for carrying or havingcomputer-executable instructions or data structures stored thereon.

In some embodiments, the different components, modules, engines, andservices described herein may be implemented as objects or processesthat execute on a computing system (e.g., as separate threads). Whilesome of the systems and methods described herein are generally describedas being implemented in software (stored on and/or executed by generalpurpose hardware), specific hardware implementations or a combination ofsoftware and specific hardware implementations are also possible andcontemplated.

In accordance with common practice, the various features illustrated inthe drawings may not be drawn to scale. The illustrations presented inthe present disclosure are not meant to be actual views of anyparticular apparatus (e.g., device, system, etc.) or method, but aremerely idealized representations that are employed to describe variousembodiments of the disclosure. Accordingly, the dimensions of thevarious features may be arbitrarily expanded or reduced for clarity. Inaddition, some of the drawings may be simplified for clarity. Thus, thedrawings may not depict all of the components of a given apparatus(e.g., device) or all operations of a particular method.

Terms used herein and especially in the appended claims (e.g., bodies ofthe appended claims) are generally intended as “open” terms (e.g., theterm “including” should be interpreted as “including, but not limitedto,” the term “having” should be interpreted as “having at least,” theterm “includes” should be interpreted as “includes, but is not limitedto,” etc.).

Additionally, if a specific number of an introduced claim recitation isintended, such an intent will be explicitly recited in the claim, and inthe absence of such recitation no such intent is present. For example,as an aid to understanding, the following appended claims may containusage of the introductory phrases “at least one” and “one or more” tointroduce claim recitations. However, the use of such phrases should notbe construed to imply that the introduction of a claim recitation by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should be interpreted to mean “at least one”or “one or more”); the same holds true for the use of definite articlesused to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitationis explicitly recited, it is understood that such recitation should beinterpreted to mean at least the recited number (e.g., the barerecitation of “two recitations,” without other modifiers, means at leasttwo recitations, or two or more recitations). Furthermore, in thoseinstances where a convention analogous to “at least one of A, B, and C,etc.” or “one or more of A, B, and C, etc.” is used, in general such aconstruction is intended to include A alone, B alone, C alone, A and Btogether, A and C together, B and C together, or A, B, and C together,etc. For example, the use of the term “and/or” is intended to beconstrued in this manner.

Further, any disjunctive word or phrase presenting two or morealternative terms, whether in the description, claims, or drawings,should be understood to contemplate the possibilities of including oneof the terms, either of the terms, or both terms. For example, thephrase “A or B” should be understood to include the possibilities of “A”or “B” or “A and B.”

Additionally, the use of the terms “first,” “second,” “third,” etc., arenot necessarily used herein to connote a specific order or number ofelements. Generally, the terms “first,” “second,” “third,” etc., areused to distinguish between different elements as generic identifiers.Absence a showing that the terms “first,” “second,” “third,” etc.,connote a specific order, these terms should not be understood toconnote a specific order. Furthermore, absence a showing that the termsfirst,” “second,” “third,” etc., connote a specific number of elements,these terms should not be understood to connote a specific number ofelements. For example, a first widget may be described as having a firstside and a second widget may be described as having a second side. Theuse of the term “second side” with respect to the second widget may beto distinguish such side of the second widget from the “first side” ofthe first widget and not to connote that the second widget has twosides.

All examples and conditional language recited herein are intended forpedagogical objects to aid the reader in understanding the invention andthe concepts contributed by the inventor to furthering the art, and areto be construed as being without limitation to such specifically recitedexamples and conditions. Although embodiments of the present disclosurehave been described in detail, it should be understood that the variouschanges, substitutions, and alterations could be made hereto withoutdeparting from the spirit and scope of the present disclosure.

The invention claimed is:
 1. A method to present communications, themethod comprising: captioning, by a transcription system during a callbetween a first user using a first captioning telephone device and asecond user using a second telephone device, words in audio spoken bythe second user into the second telephone device, the captioninginvolving a human assistant; presenting the captioned words on a firstdisplay of the first captioning telephone device with a first indicatorthat indicates that the captioned words were spoken by the second user;receiving text typed into the second telephone device by the seconduser; and presenting the received text on the first display of the firstcaptioning telephone device with a second indicator that indicates thatthe received text was typed by the second user.
 2. The method of claim1, wherein: the first user is a hearing-impaired user; and the seconduser is a hearing user.
 3. The method of claim 1, wherein: the secondtelephone device comprises a smartphone device having a touchscreendisplay and running a captioning app; and the text typed into thesmartphone device comprises text typed into the captioning app via thetouchscreen display.
 4. The method of claim 3, further comprising:receiving second text typed into the first captioning telephone deviceby the first user; and presenting the second text in the captioning appon the touchscreen display of the smartphone device.
 5. The method ofclaim 1, wherein: the second telephone device comprises a Voice-over-IP(VoIP) telephone device having a second display and running a captioningapp; and the text typed into the VoIP telephone device comprises texttyped into the captioning app.
 6. The method of claim 5, furthercomprising: receiving second text typed into the first captioningtelephone device by the first user; and presenting the received secondtext in the captioning app on the second display of the VoIP telephonedevice.
 7. The method of claim 1, wherein: the second telephone devicecomprises a dual-tone multi-frequency signaling (DTMF) telephone deviceincluding a telephone keypad and a speaker; and the text typed into theDTMF telephone device by the second user comprises text typed into thetelephone keypad.
 8. The method of claim 7, further comprising:receiving second text typed into the first captioning telephone deviceby the first user; converting the second text into second audio using atext-to-speech conversion; and sending the second audio to the speakerof the DTMF telephone device.
 9. The method of claim 1, furthercomprising: presenting the received text on a display of a computingdevice of the human assistant.
 10. At least one non-transitorycomputer-readable media configured to store one or more instructionsthat in response to being executed by at least one computing systemcause performance of the method of claim
 1. 11. A captioning telephonedevice comprising: a display; a speaker; a processor communicativelycoupled to the display and to the speaker; and at least onenon-transitory computer-readable media communicatively coupled to theprocessor and configured to store one or more instructions that, whenexecuted by the processor, cause the captioning telephone device toperform operations comprising: receiving, from a transcription systemduring a call between a first user using the captioning telephone deviceand a second user using a second telephone device, captions of words inaudio spoken by the second user into the second telephone device, thegeneration of captions by the transcription system involving a humanassistant; presenting the captioned words on the display with a firstindicator that indicates that the captioned words were spoken by thesecond user; receiving text typed into the second telephone device bythe second user; and presenting the received text on the display with asecond indicator that indicates that the received text was typed by thesecond user.
 12. The captioning telephone device of claim 11, wherein:the first user is a hearing-impaired user; and the second user is ahearing user.
 13. The captioning telephone device of claim 11, wherein:the second telephone device comprises a smartphone device having atouchscreen display and running a captioning app; and the text typedinto the smartphone device comprises text typed into the captioning appvia the touchscreen display.
 14. The captioning telephone device ofclaim 13, wherein: the captioning telephone device further comprises atelephone keypad; the operations further comprise receiving second texttyped into the telephone keypad of the captioning telephone device bythe first user; and the operations further comprise sending the secondtext for presentation in the captioning app on the touchscreen displayof the smartphone device.
 15. The captioning telephone device of claim11, wherein: the second telephone device comprises a Voice-over-IP(VoIP) telephone device having a second display and running a captioningapp; and the text typed into the VoIP telephone device comprises texttyped into the captioning app.
 16. The captioning telephone device ofclaim 15, wherein the operations further comprise: receiving second texttyped into the captioning telephone device by the first user; andsending the second text for presentation in the captioning app on thesecond display of the VoIP telephone device.
 17. The captioningtelephone device of claim 11, wherein: the second telephone devicecomprises a dual-tone multi-frequency signaling (DTMF) telephone deviceincluding a telephone keypad and a second speaker; and the text typedinto the DTMF telephone device by the second user comprises text typedinto the telephone keypad.
 18. The captioning telephone device of claim17, further comprising: receiving second text typed into the captioningtelephone device by the first user; converting the second text intoaudio using a text-to-speech conversion; and sending the audio to thesecond speaker of the DTMF telephone device.
 19. A method to presentcommunications, the method comprising: obtaining, from a transcriptionsystem, a transcription of audio spoken by a first user into a firstdevice during a communication session between the first user using thefirst device and a second user using a second device; presenting thetranscription on a display of the first device with a first indicatorthat indicates that the transcription was spoken by the second user;obtaining text typed into the second device by the second user; andpresenting the obtained text on the display of the first device with asecond indicator that indicates that the obtained text was typed by thesecond user.
 20. The method of claim 19, further comprising directingthe obtained text from the second device to the transcription system.